Skip to content

feat(ogc): add the water well field operations internal layer - #914

Merged
ksmuczynski merged 5 commits into
stagingfrom
kas-water-well-field-operations-ogc-layer-bdms-1202
Sep 1, 2026
Merged

feat(ogc): add the water well field operations internal layer#914
ksmuczynski merged 5 commits into
stagingfrom
kas-water-well-field-operations-ogc-layer-bdms-1202

Conversation

@ksmuczynski

@ksmuczynski ksmuczynski commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds water_well_field_operations, a new internal-only OGC feature layer. This layer contains information a field crew needs to plan and run a visit in one place, instead of having to manually join information from the existing layers.

Also includes supporting documentation:

  • Column-by-column sources: docs/water-well-field-operations-columns.md
  • Design rationale: docs/water-well-field-operations-layer.md

Why

This PR addresses the following problem / context:

  • Existing layers were missing information necessary for planning, executing, and reviewing field events.
  • Users had to manually join multiple layers together in ArcGIS or QGIS to review operational information.
  • This new field operations layer addresses those gaps.

How

Implementation summary - the following was changed / added / removed:

  • Added a two-relation migration: (1) ogc_internal_water_well_field_operations, a view pygeoapi serves directly, and (2) a nightly-refreshed materialized view that supplies its expensive statistics (manual water-level counts, chemistry-sampling counts, and continuous-reading counts). Status and permission columns stay on the pygeoapi view, so a permission that expires this morning shows as expired immediately.
  • Published construction and location detail. Every well gets one row with identity, location, construction details, completion formation, aquifer system, and current measuring point.
  • Published status, permissions, equipment, contacts, and notes.
    • Four status types: well, monitoring, open, datalogger suitability.
    • Three standing landowner permissions, each true, false, or null, where null means nobody's recorded one yet.
    • Monitoring frequency and group membership.
    • Manual water-level, chemistry, and field-visit statistics.
    • Every sensor currently installed, including cameras and barometers. has_datalogger remains available as a logger-specific flag.
    • Landowner contacts.
    • A column per note type, 13 total, including access, directions, and maintenance.
  • Made current-record columns date-aware. Status, permission, measuring point, and monitoring frequency reflect what's true today, so an expired permission or a closed-out status doesn't linger past its end date.
  • Added an index for every join this layer needs.
  • Emitted multi-valued columns (screens, purposes, casing materials, equipment, notes) as delimited text. Crews pull this layer into ArcGIS Pro/QGIS and export it to File Geodatabase or GeoPackage, and neither format has a list type.

Example of the view's fields and values

Field Value
id 337
name Example Well No. 12
station_type water well
release_status public
alternate_ids NMOSE: RG-12345
latitude 35.1
longitude -106.65
elevation 1800.0
well_depth 250.0
hole_depth 260.0
well_casing_diameter 6.0
well_casing_depth 240.0
well_completion_date 2010-05-01
well_driller_name Acme Drilling
well_construction_method null
well_pump_type null
well_pump_depth 200.0
formation_completion_code Example Formation XYZ
formation_completion_description Fractured basalt aquifer, mid-Miocene
well_purpose Domestic, Irrigation
well_casing_material PVC, Steel
aquifer_system_name Example Aquifer A
screen_count 2
screen_depth_top 120; 210
screen_depth_bottom 140; 230
screen_description Upper zone, alluvial gravel; Lower zone, fractured basalt
mp_height 2.3
mp_description New steel collar, north side
well_status Active, pumping well
monitoring_status Currently monitored
open_status Open
datalogger_suitability_status Datalogger can be installed
may_measure_water_level null
may_sample_water_chemistry false
may_install_datalogger true
permission_granted_by null
monitoring_frequency Quarterly
group_names South Valley Monitoring Network
group_types Monitoring Plan
manual_water_level_count 0
manual_water_level_first_date null
manual_water_level_last_date null
days_since_manual_water_level null
last_depth_to_water_ft null
chemistry_sample_count 0
chemistry_sample_last_date null
days_since_chemistry_sample null
field_event_count 2
date_last_visited 2023-11-02
has_datalogger true
datalogger_deployment_count 1
sensor_type Camera; Pressure Transducer
model Reconyx HC600; Rugged Troll 200
serial_no RX-9001; RT200-001
sensor_status In Service; In Service
installed_date 2023-03-01; 2022-06-01
recording_interval ; 15
recording_interval_units ; minute
hanging_point_desc Strapped to fence post, facing wellhead; Top of steel collar
continuous_reading_count 0
continuous_first_datetime null
continuous_last_datetime null
days_since_continuous_reading null
contact_count 1
primary_contact_name Jane Rancher
primary_contact_organization null
primary_contact_role Owner
primary_contact_type Primary
primary_contact_phone 575-555-0110
primary_contact_email jane@example.com
contact_names Jane Rancher
access_notes Gate code 4471. Owner prefers calls after 8am.
directions_notes Turn off Hwy 380 at mile marker 12, well is 0.3mi north.
communication_notes null
construction_notes null
maintenance_notes Pump replaced 2024
historical_notes null
general_notes null
water_notes Well produces slightly sulfurous water
water_quality_notes null
sampling_procedure_notes null
coordinate_notes null
owner_comment_notes null
site_notes_legacy null
point POINT(-106.65 35.1)

Notes

Any special considerations, workarounds, or follow-up work to note?

  • Internal-only, with no public counterpart, and none is planned. This layer publishes landowner contact details and staff-written notes, so every credential accepted by the internal mount now reaches personal information.
  • Nightly refresh means the statistics columns can be up to a day stale. Fine if crews plan the next day's route the evening before; worth watching if that assumption changes.

jirhiker and others added 3 commits September 1, 2026 10:37
Adds water_well_field_operations, an OGC API - Features collection on
/ogcapi-internal only: one feature per water well carrying what a field crew
needs to plan and execute a visit. Every other well layer answers a scientific
question -- water level, chemistry, trend. This one answers an operational one:
may we go to this well, what are we allowed to do when we get there, and is it
overdue?

86 columns: identity and location (including latitude and longitude in decimal
degrees and how the elevation was obtained), construction detail, the current
measuring point, all five status types, the three standing landowner
permissions, monitoring frequency and group membership, manual water-level and
chemistry-sampling statistics, field-visit history, the installed logger and
its continuous record, landowner contacts, and access and directions notes.

Two relations rather than one (migration e1f2a3b4c5d6):

  ogc_internal_water_well_field_operations_stats  materialized, nightly refresh
      Aggregates over observation and transducer_observation.

  ogc_internal_water_well_field_operations        plain view, served by pygeoapi
      Live join of thing, location, status_history, permission_history,
      measuring_point_history, monitoring_frequency_history, deployment,
      sensor, contact and notes, LEFT JOINed to the stats matview.

The split is not tidiness. Staleness is dangerous on exactly the columns that
are cheap to read: a revoked sampling permission that still reads true until
the next nightly refresh sends a crew onto land they are no longer welcome on.
The current-record rule is also written against CURRENT_DATE, which inside a
materialized view would freeze at refresh time.

Three decisions worth review:

* Internal-only with no public twin. The layer publishes landowner and operator
  contact details -- name, organization, role, phone, email -- alongside
  staff-written access notes. ogc_water_well_field_operations does not exist
  and must never be created. Every credential the internal mount accepts now
  reaches personal information, the static desktop-GIS keys included; notices
  added to CLAUDE.md and docs/internal-ogc-desktop-gis.md.

* The three permission columns are three-valued. NULL means no permission is on
  record, which is not a refusal -- it means nobody has asked the landowner
  yet. Collapsing it to false would tell a crew a well is off limits when the
  truth is unknown.

* History reads honour end_date, unlike ogc_actively_monitored_wells, which
  takes the greatest start_date and lets a status closed in 2019 read as
  current. Written up in docs/ogc_conventions.md.

Requested as amp_field_well_operations and renamed before merge per the
pre-merge check in docs/ogc_conventions.md: amp is an unexpanded abbreviation
whose expansion is recorded nowhere in this codebase, AMP labels the layer's
audience rather than filtering its rows (the row set is every water well), and
Group B analytic layers prefer a water_well_ prefix. The AMP crews stay named
in the collection description.

Multi-valued columns are comma-joined text rather than arrays: this layer is
pulled into ArcGIS Pro and QGIS and exported to File Geodatabase and GeoPackage
for offline field use, and neither format has a list type.

Depth to water uses the (value - COALESCE(measuring_point_height, 0))
convention shared with ogc_water_well_summary, ogc_latest_depth_to_water_wells
and ogc_well_water_column, and elevation_method reuses that summary's
provenance lookup verbatim, so the layers cannot disagree.

Twelve supporting indexes come with the migration; none of the foreign keys
these lookups traverse was indexed.

Design rationale in docs/water-well-field-operations-layer.md, column-by-column
sources in docs/water-well-field-operations-columns.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independently-drafted internal water-well layers existed after a
side-by-side field comparison: feat/amp-field-operations-ogc-layer
and a competing draft, kas-water-well-operations-ogc-layer-
bdms-1202. Rather than ship both under colliding migration IDs, this
branch (kas-water-well-field-operations-ogc-layer-bdms-1202) started
from feat/amp-field-operations-ogc-layer. It already had tests,
indexes, docs, and correct date-window current-record semantics. The
agreed changes from the comparison follow:

- Dropped nma_pk_welldata, county, state, quad_name, elevation_method,
  nma_formation_zone, and measuring_point_start_date. None of them
  serve this layer's operational purpose.
- Dropped every `_since`/`_reason` column (one per status type, plus
  monitoring_frequency_since). The current value itself (well_status,
  monitoring_status, monitoring_frequency, etc.) stays; only the
  history around it was dropped.
- Renamed thing_type, well_casing_materials, well_purposes,
  measuring_point_height/description, and field_event_last_date to
  match the naming already established on the public thing-type
  views (station_type, well_casing_material, well_purpose,
  mp_height/mp_description, date_last_visited).
- Added formation_completion_description and aquifer_system_name,
  ported from the competing draft.
- Restored well_screen to full per-interval detail (screen_depth_top,
  screen_depth_bottom, screen_description), ported from the competing
  draft, instead of the count-plus-min/max summary this branch
  shipped with. A driller doing rehab work needs the actual screened
  intervals, not an overall range.
- Broadened "currently installed equipment" from logger-only to any
  currently installed sensor, so a camera or barometer is no longer
  invisible. has_datalogger/datalogger_deployment_count stay
  logger-scoped for whoever needs that narrower signal.
- Published a column per remaining notes.note_type value (13 total),
  extending the existing access_notes/directions_notes pattern.
- Dropped access_status, since status_value has no lexicon terms
  scoped to Access Status. The column could only ever read NULL, and
  access_notes already covers the same need.
- Wrapped every aggregated equipment and well_screen expression except
  sensor_type in COALESCE(..., '') before string_agg, so a row with a
  null value in one field (e.g. a camera with no recording_interval,
  or a screen with no recorded description) leaves an empty segment
  instead of silently shortening that column's list and breaking
  alignment with the others.

Config, field descriptions, design docs, and tests updated to match.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Coverage

79.35% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
services/materialized_views.py 1 0 100%
TOTAL 1 0 100%

tests/features/ogc-cleanup-sprint1.feature's "Internal database
relations are separate from public relations" scenario asserts that
every ogc_internal_ relation in the schema has a distinct ogc_ public
counterpart. That held for every internal-only collection so far
(locations, avg_tds_wells, latest_depth_to_water_wells): each is
hidden from the public collections catalog, but both relations still
exist in the database.

water_well_field_operations breaks that pattern on purpose.
ogc_water_well_field_operations does not exist at all, by design (see
docs/water-well-field-operations-layer.md section 3), since the
layer publishes landowner contact details and staff-written notes.
The check had no way to know that was intentional, so it failed CI on
this PR.

Added an explicit exception set for the two relations this layer
introduces, rather than loosening the assertion generally. The check
should still fail if some future ogc_internal_ relation is missing
its public twin by accident.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new view’s multi-value aggregations and permission_granted_by semantics have verified correctness issues that can misalign positional fields and/or contradict documented meaning.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new internal-only OGC API Features collection (water_well_field_operations) to serve field crews with an operational “one-stop” well layer (construction + current status/permissions + installed equipment + contacts + notes), backed by a view plus a nightly-refreshed stats materialized view.

Changes:

  • Adds Alembic migration creating ogc_internal_water_well_field_operations (view) and ogc_internal_water_well_field_operations_stats (matview) plus supporting indexes.
  • Publishes the collection on /ogcapi-internal (pygeoapi internal config) and documents every field in core/ogc-field-descriptions.yml.
  • Adds comprehensive tests and supporting documentation describing design rationale, column lineage, and internal-only/PII implications.
File summaries
File Description
tests/test_pygeoapi_mount.py Marks the new collection as internal-only in mount-related tests.
tests/test_ogc.py Ensures the collection is excluded from the public OGC collections list.
tests/test_ogc_water_well_field_operations.py New test suite validating the layer’s load-bearing semantics and internal-only mounting.
tests/test_cli_commands.py Updates expected matview refresh list/count for CLI refresh command.
tests/features/steps/ogc-cleanup-sprint1.py Exempts the new internal-only-without-public-twin relations from parity checks.
services/materialized_views.py Registers the stats matview for refresh via the CLI path.
docs/water-well-field-operations-layer.md Design doc explaining scope, PII decision, current-record semantics, and wiring.
docs/water-well-field-operations-columns.md Column-by-column source mapping for the new view/matview.
docs/ogc_conventions.md Documents naming and “current-record” semantics conventions for the new layer.
docs/internal-ogc-desktop-gis.md Adds a prominent notice that internal OGC credentials now convey PII due to this layer.
core/pygeoapi-config-internal.yml Publishes the new collection on the internal pygeoapi mount.
core/ogc-field-descriptions.yml Adds per-field titles/descriptions for the new collection.
CLAUDE.md Records the internal-PII nature of /ogcapi-internal and this layer for future changes.
alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py Implements the stats matview + feature view and supporting indexes.
Review details

Suppressed comments (2)

alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py:533

  • screens has the same positional-alignment requirement as installed_equipment, but orders all three string_agg() columns only by screen_depth_top. If two screens share the same top depth, tie ordering can differ between aggregates and misalign bottom/description segments. Add a stable tie-breaker (e.g., ws.id) to the ORDER BY in all three aggregates.
                string_agg(
                    COALESCE(ws.screen_depth_top::text, ''), '; '
                    ORDER BY ws.screen_depth_top NULLS LAST
                ) AS screen_depth_top,
                string_agg(

alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py:781

  • group_names and group_types are intended to be index-aligned, but both string_agg() calls order only by group_name. If two groups share the same name, tie ordering is undefined and the names/types can become misaligned between the two aggregates. Add a deterministic tie-breaker (e.g., group_id).
                string_agg(dm.group_name, ', ' ORDER BY dm.group_name)
                    AS group_names,
                string_agg(dm.group_type, ', ' ORDER BY dm.group_name)
                    AS group_types
  • Files reviewed: 14/14 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@ksmuczynski
ksmuczynski merged commit 6fd64fa into staging Sep 1, 2026
9 checks passed
@ksmuczynski
ksmuczynski deleted the kas-water-well-field-operations-ogc-layer-bdms-1202 branch September 1, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants